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HANDWRITING USER INTERFACE FOR 



PERSONAL DIGITAL ASSISTANTS AND THE LIKE 



BACKGROUND OF THE INVENTION 



FIELD OF THE INVENTION 



The present invention is related to personal digital assistants (PDAs) and more 
particularly to a user input interface for a PDA or the like. 

BACKGROUND DESCRIPTION 

Portable computing devices, such as what is normally referred to as a personal 
digital assistant (PDA), are increasing in popularity. A typical PDA is a limited 
function microcomputer provided with a pressure sensitive liquid crystal diode (LCD) 
display (a touch pad or a touch screen) for input and output (I/O). PDAs are being 
adapted for wireless Internet communication, e.g., using a modem for e-mail and web 
browsing. Further, for text input PDAs are known that have a specialized stroke 
based alphabet interface, e.g., Graffiti®, a selectable on-screen QWERTY keypad, or 
an expansion pack keyboard. 

As these portable devices become smaller and more specialized, text input has 
become more difficult and less practical. Typical prior art handwriting recognition 
software may require users to learn special characters or effect a handwriting style in 
order to enter text. Text input using the Graffiti® unistroke (i.e., written with a single 
pen trace) alphabet can be un-natural because it requires users to adhere to strict rules 



that restrict character shapes; text input using an on-screen QWERTY keypad is 
somewhat clumsy because only small reductions in size can be made to keyboards 
before they become awkward to use. An expansion keyboard is impractical for on- 
the-go input. With either, the tapping on individual characters or the typing is less 
desirable than being able to handwrite notes or messages. Meanwhile, the demand for 
PDA information exchange, e-mail and internet access requires entry and retrieval of 
increasing amounts of data with the handheld device. 

Natural handwriting recognition (HWR) programs have been developed to add 
to function and usefulness to PDAs and are crucial to the growth of mobile computing 
in the communications field. To use such handwriting recognition software, such as 
Transcriber (formerly known as Calligrapher) from Microsoft Corporation, the user is 
allowed to write a message anywhere on the screen, i.e., on top of any displayed 
application and system elements. Once the text is corrected, it may be embedded in 
the e-mail message, for instance, and, the next sentence or string of words can be 
entered. However, typically, correction is deferred until the message or word storing 
is completed, when the whole string may be displayed, because during input the entire 
display is used for handwriting input. 

However, these write anywhere approaches require a special mechanism to 
distinguish pen movement events that correspond to handwriting input from pen 
events that are intended to manipulate user interface elements such as buttons, scroll 
bars and menus. Often it is difficult to differentiate between these two modes of 
stylus operation, viz. that of a writing implement for text entry (inking mode) and its 
control function such as for clicking on application icons and the like (control mode). 



Another problem with a write-anywhere user interface is that fingers, as the writer is 
moving his/her hand through the screen, can often interfere with the (pressure-based) 
pen tracking mechanism. Simultaneous pressure from the stylus and a carelessly 
positioned pinky finger for instance can cause the device to mislocate the intended 
stylus entry point, e.g., the device may use the average of the two contact locations. 
These shortcomings can lead to text input errors and the attendant aggravation and 
input delays caused by such errors. 

Typically, these state of the art handwriting recognition programs exhibit a 
top-n recognition accuracy of around 85% (top-1) and 95% (top-5), where top-n 
recognition accuracy is a measure of how often the correct answer is among the 
highest ranked n results. An 85% recognition accuracy, where one in six words is 
misrecognized, is not particularly tolerable for many users. However, at higher 
recognition rates of 95% and above, where only 1 in 20 words is misrecognized, users 
are more inclined to use handwriting recognition software. Therefore, easy access to 
recognition alternates — i.e., top-n results, is a very important HWR feature. 

An additional user interface issue with a write-anywhere text input paradigm 
is that there are usually no input method control elements visible anywhere on the 
screen. For instance, access to recognition alternates might require a special pen 
gesture. As such, a write-anywhere interface generally is not very appealing to less 
advanced users. Furthermore, recognition in the write-anywhere case is more difficult 
because there is no implicit information provided to the recognition engine regarding 
word separation, orientation, or size of the text. 
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Thus, there is a need for handwriting input user interface that easily 
distinguishes between control mode and inking mode of the pen, that allows easy 
access to recognition alternatives, and that enables accurate recognition of 
handwritten words. 

BRIEF DESCRIPTION OF THE DRAWINGS 

The foregoing and other objects, aspects and advantages will be better 
understood from the following detailed preferred embodiment description with 
reference to the drawings, in which: 

Figure 1 shows a preferred embodiment handheld device with a graphical 
handwriting user interface according to preferred embodiment of the present 
invention; 

Figure 2 shows an example of the handwriting user interface (HUI) displaying 
a word correction keyboard for manually correcting a handwritten word; 

Figure 3 is a flow diagram of an example of a method for implementing the 
handwriting user interface of the preferred embodiment of the present invention. 




DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS 

The present invention is a method of interfacing with and a handwriting user 
interface (HUI) for small (pocket-shirt sized) portable devices with a touch-enabled 
input/output (I/O) screen, such as are commonly known as personal digital assistants 
5 (PDAs). The portable devices may be capable of wireless message transmission (such 
as for web browsing and/or e-mail). The user interface of the present invention is 
typically in software and loaded into PDA storage. A state of the art handwriting 
recognition engine also is included in software. The handwriting user interface of the 
present invention enhances the usability, flexibility and power of the handheld device 
10 in which it is installed. An entire message may be quickly handwritten, converted, 
stored and then, transmitted, for example. 

Figure 1 shows a preferred embodiment pocket sized handheld device 100 
with a housing 101 graphical handwriting user interface 102 according to preferred 
embodiment of the present invention. A lower portion of the display is designated as 

15 a handwriting input area 104. In the illustrated and preferred form, the housing 101 
has a compact pocket sized form with a rectangular configuration having dimensions 
on the order of 120 x80 mm with the screen height being 82 mm and the width being 
62 mm, for example. Action icons 106, 108, 110, 112 and 114 are disposed at a right 
side of the handwriting user interface 102. Recognized text is displayed at the top of 

20 the screen under a file management tool bar 1 16. In this embodiment, a scroll bar 118 
is disposed at the right side of the interface display 102. As each word is recognized, 
it is shown inserted into the text at the top of the interface display 102 and, a 
secondary list of potential recognition candidates may be displayed in a box 120 and 
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offered for substitution for or in lieu of the recognized word. Although the secondary 
word list box 120 is preferably displayed in the input area 104, in this example it is 
shown just above the handwriting input area 104. 

Handwritten Entries are made at the designated input area 104 on the touch 
screen, preferably of dimensions 0.30*H by W, where H and W are the height and 
width of the device screen. The preferred location of the input area is the bottom of 
the screen 102, so as to only partially block view of any application currently running 
on the device. Thus, only the stylus is above the input area 104 during entry with the 
largest portion of the hand resting on the housing 101 therebelow. Handwritten words 
are entered into the input area 104 one word at a time using a stylus and, recognition 
results can be displayed in the same input area 104 or in the normal display area of the 
screen above the input area 104. 

The device 100 may include a communications function and, to that end in this 
embodiment, an antenna 122 is shown at the top of the device 100. Individual 
function switches, buttons and other controls are disposed about the device, as is 
deemed appropriate for the particular device. The device 100 may also include an 
expansion port 124 or an expansion port function may be provided wirelessly through 
antenna 122. Preferably, the device 100 runs under a state of the art operating system 
for such handheld devices, e.g. Windows® CE from Microsoft Corporation, Epoc® 
from Symbian or the Palm OS® from Palm, Inc. 

The preferred embodiment HUI of the present invention employs a 
handwriting recognition engine capable of recognizing handwritten words, written 



using any combination of writing styles (i.e., cursive, print, and mixed) to improve 
throughput on text entry, as it allows a more natural writing style to be employed 
versus, for example, engines that require only individual characters requiring a pause 
between each as it is entered. Preferably, the recognition engine is the QuickPrintPro 
engine from Motorola, Inc., Lexicus division. The recognition engine typically 
includes a main dictionary and a user dictionary to which the user may add words to 
supplement the main dictionary. The recognition engine compares a handwritten 
input word against all words contained in the main dictionary and the user dictionary. 
A probability score is generated by the recognition engine for each dictionary word 
which is indicative of the likelihood that the handwritten word matches that particular 
dictionary word. Based on each words' probability score, a list of likely matches is 
collected. 

From the recognition results, the handwriting recognition engine calculates a 
confidence level for the one word (the primary word) with the highest probability. If 
that confidence level exceeds a preselected or confidence threshold, it is taken as an 
indication that the word with the highest probability is in fact correct and the highest 
scoring word is displayed as the primary word choice. All other results are referred to 
as secondary word choices and may be included in the pop-up list in box 120. So, if 
the confidence level is above the preselected threshold, the HUI automatically loads a 
primary word choice into the device's input buffer for delivery to the active 
application. Otherwise, when the confidence level of the primary word choice is 
below the confidence threshold, an indication is provided that the recognition engine 
cannot find a likely candidate, e.g., displaying "???" or something similar into the 



device's input buffer. The primary and secondary word choice may be displayed in 
the pop-up list box 120. 

The number of words, n, listed in the pop-up list 120, is user selectable and 
generally is small enough (5) that the pop us list is contained within the input area 
104. Providing ready access to word recognition results in the pop-up list increases 
the likelihood that the correct word is, at the very least, included in the group of top-n 
words in box 120. Thus, the word recognition rate is generally higher for the group 
than the overall individual word recognition rate. The word group recognition rate 
improvement may be as much as 10%. Therefore, presenting the top-n results to the 
user, where n is typically 5, improves the likelihood that the correct word is displayed, 
even if the correct word is not the top scoring entry. The correct word may be 
selected from the group nearly as quickly as accepting a correctly recognized word. 
Each newly selected word choice is loaded into the system input buffer, and the 
previously misrecognized word, if any, is deleted from the buffer. 

Action icons 106, 108, 110, 112, 114 are displayed to provide virtual buttons 
for editing any previously entered text. Preferably, the icons are displayed together at 
any side of the input area (e.g., left, right, top or bottom). Editing operations may 
include, but are not limited to: insert a space 108, backspace 112, delete 114, 
capitalize recognition result 110, and undo insertion of last recognition result 106. 
Further, as each word is entered and recognized, a stylus may be used to select one or 
more characters of the word in a text field of the active application. The preferred 
recognition engine is also capable of recognizing individual stand-alone characters. 
At any time, the user can select one (or more) character(s) from a previously entered 



word and write a new character(s) in the input area with the result replacing the 
selected text. Optionally, the editing icon 106 can automatically select a correction 
keyboard which may be used to edit the last recognition result. When selected, the 
correction keyboard is displayed in the input area 102. 

Figure 2 shows an example of the preferred embodiment HUI displaying a 
word correction keyboard for manually correcting a handwritten word recognition 
result. In this mode, the user interface displays a QWERTY keyboard 132 in the 
input area and a word correction window 134. The previously input text is displayed 
at the top of the screen. Each word is entered and the last recognition result remains 
displayed for editing in the editing area. As noted above, a single word can be 
selected or, individual letters within the word may be selected and corrected using the 
QWERTY keyboard 132. A special purpose key or button 136 may be included in the 
correction keyboard 132 for inserting the corrected word or substitute word into the 
user dictionary for inclusion in subsequent recognition. 

Additional icons (not shown) may be included on the display to allow the user 
to change selected configuration settings. Configuration settings may include 
handwriting style preferences and recognition options. Typical recognition options 
may include an option to propose upper-case at the beginning of a word, an option to 
suggest end of word punctuation, the number of recognition results displayed in the 
pop-up list, the location of editing buttons (i.e., left or right hand side of the input 
area), and user dictionary maintenance, i.e., viewing, adding, and/or deleting entries. 
The option to propose upper-case may be such that, if set, the recognition engine 
attempts to recognize the input with and without a leading upper-case letter. The 



option to suggest punctuation also may be included such that the recognition engine 
may be directed to recognize punctuated handwritten input, automatically discerning 
when trailing punctuation marks are included. Punctuation mark recognition is 
simpler in the context of a word. A period, for instance, written by itself is merely 
5 meaningless and could be interpreted as anything. However, small digital ink point at 
the end of a word is much easier to identify and classify as a punctuation mark, e.g. a 
period, comma, etc. 

Handwritten input entry may be provided in unrestricted mixed style that 
includes cursive (i.e., contiguous characters in each word touching or connected), 

10 pure print (i.e., characters in every word disconnected and do not touch), pseudo-print 
(at most pairs of characters in words touch) or any combination thereof. Thus, for 
mixed entry, the user is not restricted to cursive, print or pseudo-print inputs. 
However, to facilitate recognition accuracy and entry speed, the user may designate 
that entry is to be in one mode only, i.e., cursive, pure print or pseudo print. By thus 

15 designating entry mode, the number and complexity of created character alternatives 
possible may be reduced for the handwriting recognition engine, increasing both 
recognition accuracy and speed. 

Single word-at-a-time input recognition is advantageous over character-at-a- 
time recognition for text input in these kind of devices, because it enables higher 
20 writing throughput when composing messages. Further, single word input in the 
designated input area is more desirable than writing multiple words or sentences 
anywhere on the screen, for example, because it is much more structured, simpler to 
use and, therefore, leads to more predictable and consistent results. Recognition 
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errors are avoided that could otherwise result from segmenting an input string into 
words and from corresponding conflicts. These errors and conflicts also result from 
the inherent ambiguity of inputting with a single pointing device, i.e., a stylus, 
wherein the stylus is used both as an inking pen for writing and, as a mouse-type 
5 pointing device for function selection. For example, the device must distinguish 
between an inking stroke and scrolling the screen by dragging the stylus. By 
designating an input area for writing, such conflicts are resolved simply: the stylus 
functions as an inking pen inside the writing area and as a non-inking pointing 
device/mouse outside of the input area. 

10 Figure 3 shows a flow diagram of an example of a method 140 for 

implementing the handwriting user interface of the preferred embodiment of the 
present invention. First, in step 142 a handwritten word is entered into the designated 
screen input area. In step 144 a check is made to determine when the handwritten 
^entry-is-complete; this is typically done with a timer, by pressing a space key or by a 



15 (spec^ When the handwritten entry is complete, continuing to step 146, 

the handwriting recognition engine matches the handwritten input against words in 
the system dictionary as supplemented by the user dictionary. In step 148 a 
confidence score is attached to the top scoring word. In step 150, the highest scoring 
words are selected from the dictionaries and displayed in the pop-up list 120. 

20 In step 152 a confidence level for the top scoring word is checked to determine 

if it exceeds the confidence threshold and so, scores high enough to be accepted as a 
positive indication of having identified the handwritten word. So, if the confidence 
level is high enough in step 154, then, it is inserted in the input buffer as primary 
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word choice for that handwritten word. In step 156 the user is allowed to decide 
whether the primary word is correct and, if so, returning to step 142, the user can 
enter a next word. If, in step 152, the confidence level is not high enough, then in step 
158, the user is prompted with an indication that the recognition result is less reliable; 
in the preferred embodiment, this indication is in the form of a special question mark 
string ("???") which is inserted in the input buffer, but it could be an audible signal, or 
any other suitable indication. 

In step 160, the pop-up list provided to the user includes the primary word, if 
any, as the top choice along with the next n-1 highest scoring words so that the user 
may examine the n highest scoring words. If the correct word is included in the pop- 
up list, then continuing to step 162, the user can select the correct word. In step 164 
that selected word is inserted into the text stream, either to replace the previously 
provided primary word or as an original word replacing the "???" string and returning 
to step 142, the user is allowed to enter a next word. 

However, if in step 160, the correct word is not listed in the pop-up list then, 
in step 166, the user is allowed to undo the entry. If the user selects to undo the entry, 
then, in step 168 the previously recognized primary word or the "???" string is 
removed from the device's input buffer and so from the display; and, returning to step 
142 the user is allowed to enter a next handwritten word. However, if in step 166, the 
user selects not to undo the previous word, then again returning to step 142, the user 
can enter a next word. Note that the HUI communicates with the currently active 
application through the device's input buffer. 
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Thus, as can be readily appreciated the HUT of the present invention provides 
a simple to use, yet elegant handwriting interface for pocket sized devices such as 
PDAs and the like. 

While the invention has been described in terms of preferred embodiments, 
those skilled in the art will recognize that the invention can be practiced with 
modification within the spirit and scope of the appended claims. 
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